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X.  IMTRODUCTION 


Trident  CCSMA  has  requested  the  Naval  Postgraduate  School  to  evaluate 
the  Department  of  the  Navy's  Tactical  Digital  Systems  Documentation 
Standard  SEGNAVINST  3560.1  dated  8  August  1974,  hereafter  referred  to  as 
3560.1,  with  respect  to  its  applicability  and  usefulness  for  softw2u:e 
maintenance.  This  report  is  divided  into  the  following  sections: 

-  Brief  Description  of  3560.1.  This  section  is  provided  to  give  the 
reader  who  is  unfamiliar  with  3560.1  a  brief  overview  of  its 
contents . 

-  Traceability.  Since  a  major  concern  of  CCSMA  is  traceability, 
3560.1  is  evaluated  with  respect  to  its  traceability  attributes. 
Traceability  exists  when  it  is  possible  to  identify  the  parts  of 
the  software  system,  and  the  corresponding  documentation,  that 
will  be  affected  by  a  change  in  the  software  stemming  from  a 
software  error  correction  or  enhancement. 

-  Usefulness  of  3560.1  for  Supporting  Software  Metintenance .  Each 
section  of  3560.1  that  is  applicable  to  software  maintenance  is 
examined  with  respect  to  usefulness  for  maintenemce. 

-  Conclusions  and  Recommendations.  Major  conclusions  concerning 
the  adequacy  of  3560.1  for  software  maintenance  are  drawn  and 
recommendations  are  made  to  meUce  it  more  suitable  for  this 
activity. 

The  major  conclusion  of  this  report  is  that,  as  it  stands, 

3560.1  is  inadequate  for  effectively  supporting  a  software 
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maintenance  activity.  However,  with  the  improvements  that  have 
been  recommended,  3560.1  could  be  the  equal  of  recently  announced 
tactical  software  standards. 
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II.  BRIEF  DESCRIPTION  OF  3560.1 

1.  Tactical  Operational  Requirement  (TOR) 

-  Tactical  digital  system  fvmctional  requirements. 

-  Serves  as  a  basic  software  specification  for  design  and  program 
implementation . 

2.  System  Operational  Specification  (SOS) 

-  Specific  operations  desired  from  application  of  the  digital 
processor  supported  data  system. 

3.  System  Operational  Design  (SOD) 

-  Plan  for  integrated  system  program. 

-  Program  functions,  core  allocations,  subprogram  definitions  and 
interfaces,  program  data  storage  plems,  and  support  programs. 

“  I/O  channel  assignments. 

Data  to  be  exchanged  between  digital  processors  and  peripherals. 

-  Timing  requirements  for  messages. 

4.  Fiinction  Operational  Specification  (FOS) 

-  Design  of  each  separate  data  function. 

-  Specify  each  required  action  at  each  operator's  position. 

5 .  Interface  Design  Specification  (IDS) 

-  Specifications  for  interdigital  processor  message  traffic  in 
format  and  content. 

6.  Program  Performance  Specification  (PPS) 

-  Describes  performance  requirements  for  the  computer  programs 
of  the  digital  processor  system. 
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-  Baseline  for  conf igiiration  control. 

-  Controlling  document  for  procuring  agency. 

7 .  Function  Operational  Design  (FOP) 

-  Program  performance  design  in  operator  f^Jnction  terms  for 
each  operator  «tnd  peripheral  equipment. 

3 .  Program  Design  Specification  (PDS) 

Design  details  for  digital  processor  programs  in  programming 
language . 

Used  for  program  maintenance. 

9.  Progrzun  Description  Document  (FDD) 

Design  details  for  each  subprogreun. 

10 .  Data  Base  Design  (DBD) 

-  Detailed  description  of  all  data  items. 

11.  Program  Package  (PP) 

-  Card  decks,  tapes,  listings,  etc. 

12 .  Test  Plan  (TPL) 

“  Defines  the  scope  of  tests  required  to  ensure  that  the  system, 
function  and  program  meet  all  appliceible  technical,  operational, 
and  performance  specifications. 

-  Establishes  detailed  acceptance  criteria  for  the  system  and 
identifies  each  level  of  testing. 

13.  Test  Specification  (TS) 

Purpose  and  scope  of  test. 

-  Identifies  softwaure,  hardware,  and  system  to  be  tested. 

14 .  Test  Procedure  (TPP) 

Instructions  for  test  ejcecution  amd  evaluation  of  results. 
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15.  Test  Report  (TR) 


-  Describes  and  evaluates  discrepancies  between  program  design  and 
operation. 

16.  Operator's  Manual  (OM) 

Presents  procedures  for  prestandby,  operate,  monitoring,  and 
recovery  of  the  digital  processor  progrcun. 

Describes  the  minimal  operating  environment. 

17.  Program  Design  Manual  (PPM) 

-  Provides  the  theory  of  combat  direction  system  prograun  processes 
that  support  the  station  operator. 

-  By  operator  and  equipment  function,  describes  the  digital 
processor  program  logic  and  algorithms  that  produce  actions  and 
data  displays  for  the  operator. 

18 .  Command  and  Staff  Manual 

-  Provides  a  nontechnical  description  of  the  tactical  system. 

-  Addresses  the  mission,  characteristics,  employment,  capabilities 
and  limitations  of  the  tactical  system. 

19.  System  Operator's  Manual 

Sole  reference  for  individual  operator  duties  and  station 
function. 

-  Explains,  at  the  level  required  by  system  operators,  every 
control  button,  switch,  readout,  and  display  affected  by  the 
system  program. 
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III.  TRACEABILITY 


In  order  to  illustrate  the  traceability  characteristics  of  3560.1, 
TABLE  1  is  provided  to  show  the  specific  references  which  a  given  section 
of  3560.1  (e.g.,  SOS)  makes  to  the  other  section(s)  (e.g.,  TOR).  Where 
these  references  occtir,  an  ”X”  is  placed  in  the  appropriate  cell  of  TABLE 
1.  The  table  is  read  by  interpreting  the  left-hand  column  as  the  section 
in  ;^ich  a  reference  occurs,  and  the  top  row,  where  there  is  an  "X"  in 
a  cell,  to  be  a  referenced  section.  The  resultant  matrix  indicates  the 
degree  of  traceability  that  exists  in  the  standard.  That  is,  the  density 
or  sparseness  of  the  matrix  is  one  measure  of  the  existence  or  ctbsence  of 
traceability,  respectively. 

The  desired  traceability  relationship,  as  implied  ty  "TDS  Documenta¬ 
tion  Relationship,"  Figxire  2  on  page  6  of  3560.1,  is  shown  in  FIGURE  1. 
The  chart  (FIGURE  1)  was  derived  from  Figure  2  of  3560.1  by  considering 
only  those  sections  of  the  standard  that  are  concerned  with  program 
development,  design,  and  testing.  The  arrows  are  upward  pointing  to 
suggest  the  use  of  documentation  for  traceability  purposes.  For  example, 
in  FIGURE  1,  a  Test  Report  (TR)  can  be  traced  to  the  Tactical  Operational 
Requirement  (TOR).  The  actual  traceability  relationships,  as  defined  by 
TABLE  1  for  the  documents  shown  in  FIGURE  1 ,  are  shown  in  FIGURE  2 . 
Although  a  great  deal  of  traceckbility  capability  exists  in  3560.1,  a 
comparison  of  FIGURE  1  with  FIGURE  2  shows  that  actual  trace£d5ility  is 
less  than  desired  traceeUsility ,  thus  indicating  inconsistencies  between 
the  objectives  of  the  standard,  relative  to  traceed^ility,  and  the  actual 


content  of  its  various  sections 
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Although  it  is  not  the  conventional  approach,  the  argument  can  be 


made  that  not  even  the  "desired  trace6d5ility  relationship"  shovm  in 
FIGURE  1  is  complete  because  traceability,  as  it  is  shown,  depends  on  a 
long  chain  of  related  dociments.  For  example,  the  Test  Procedure  (TPR) 
should  be  related  back  to  the  Tactical  Operational  Requirement  (TOR) , 
which  is  written  in  operational  user  language,  if  the  Test  Procedure  is 
to  be  a  true  reflection  of  user  requirements.  In  other  words,  it  should 
be  easy  to  see  how  the  Test  Procedxire  meets  user  requirements,  rather 
them  trace  through  several  intervening  documents  in  order  to  discern 
this  relationship.  In  addition,  the  intervening  document(s)  could  con¬ 
tain  errors,  which  would  cause  a  distorted  view  of  how  the  TPR  is  to 
meet  user  requirements. 
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FIGURE  1.  DESIRED  TRACEABILITY  RELATIONSHIP. 
•  HISSING  IN  FIGURE  2. 


TOR 


FIGURE  2.  ACTUAL  TRACEABILITY  REUTIONSHIP  BASED  ON 
REFERENCES  AMONG  SECTIONS. 
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IV.  USEFULNESS  OF  3560.1  FOR  SUPPORTING  SOFIT^ARE  .MAINTENANCE 


Traceability,  which  was  covered  in  Section  III,  is  a  capability 
which  is  useful  for  both  software  development  and  maintenance.  In  this 
section,  a  sepcific  examination  is  made  of  the  adequacy  of  3560.1  for 
software  maintenance.  Each  section  of  3560.1  that  either  mentions  main¬ 
tenance  or  could  be  useful  for  software  maintenance  is  analyzed  below. 
Recommendations  for  improvements  in  the  coverage  of  software  maintenance 
are  noted.  Section  and  page  references  are  to  3560.1. 

TABLE  2  is  provided  to  summarize  those  sections  of  3560.1  that  are 
applicable  to  software  maintencuice .  The  table  shows  section,  section 
number  and  page  references,  purpose  of  the  section,  and  an  indication  of 
whether  the  section  is  adequate  for  software  maintenance .  Brief  remcurks 
are  given,  where  appropriate.  In  the  case  of  negative  remaurks,  expla¬ 
nations  are  provided  following  TABLE  2. 

1.  Tactical  Operational  Requirement  Section  1.5.4  Test  Programs, 
p.  1-13. 

Reference  is  made  to  any  maintenance  prograuns  that  might  be 
required.  The  paragraph  seems  to  refer  to  software  that  is  used  to 
maintain  hardware.  Specific  reference  should  be  made  to  the  need  to 
provide  software  tools  for  maintaining  software  in  addition  to  the 
software  used  for  maintaining  haurdware. 

2 .  Interface  Design  Specification  Section  1.0  Purpose,  p.  2-41. 
This  section  states:  "Upon  completion  of  program  development, 

the  Interface  Design  Specification  shall  serve  as  a  joint  life  cycle 
configuration  control  device  for  digital  processor  program  maintenamce 
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of  the  interface."  Except  for  the  section  noted  in  TABLE  2  and  the 
corresponding  explanation  of  rein2Lrks,  the  IDS  contains  good  sections 
for  supporting  software  maintenance.  This  is  due  primarily  to  the 
thoroughness  of  treatment  of  items,  such  as  signal  definitions  and  inter- 
processor  communications . 

3 .  Program  Performance  Specification  Section  1.0  Purpose,  p.  2-83. 

This  section  states:  "The  Program  Performance  Specification  shall 

describe  in  detail  all  the  operational  emd  functional  requirements 
necessciry  to  design,  test,  and  maintain  the  required  digital  processor 
program."  As  shown  in  TABLE  2  and  the  explanation  of  remarks,  there  are 
sections  of  the  PPS  which  are  redundant,  unclear,  or  require  more  empha¬ 
sis  on  validation  as  opposed  to  verification.  With  respect  to  the  last 
item,  see  the  following  section  on  the  Program  Description  Docximent, 
which  contrasts  validation  with  verification.  The  comments  in  that 
section  apply  as  well  to  the  PPS.  For  these  reasons  the  PPS  is  not 
entirely  adequate  for  software  maintenance. 

4.  Prograun  Description  Document  Section  1.0  Purpose,  p.  2-137. 

This  section  states;  "As  a  detailed  compendium  of  the  subprogram 

structure,  the  Program  Description  document  will  serve  as  the  essential 
instrument  for  subsequent  use  by  operational ,  maintenance ,  and  contractor 
personnel  diagnosing  troubles,  making  adaption  changes,  designing  and 
implementing  modifications  to  the  system,  and  in  introducing  or  adding 
new  subprogram  functions  to  the  completed  program."  Thus,  the  PDD  is 
one  of  the  major  documents  which  govern  the  conduct  of  software  mainte¬ 
nance.  As  shown  in  TABLE  2  and  in  the  corresponding  explanation  of  re¬ 
marks,  the  quality  assurance  provisions  of  the  PDD  provide  sufficient 
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emphasis  on  verifying  programs,  via  testing,  but  insufficient  emphasis 
on  validating  programs  against  tactical  requirements  (e.g.,  TOR).  As 
defined  by  Lewis:  "Verification  is  the  iterative  process  aimed  at 
determining  whether  the  product  of  each  step  in  the  software  develop¬ 
ment  cycle  fulfills  all  the  requirements  levied  by  the  previous  step: 

Does  3  fulfill  requirements  of  A?  Does  C  fulfill  requirements  of  B,  and, 
implicitly,  fulfill  requirements  of  A?  ...  .  Validation  is  essentially 
that  part  of  verification  and  validation  which  looks  back  at  the  software 
requirements  2uid  determines  through  testing  that  they  are  (or  are  not) 
satisfied  by  the  observable  system  performance  indicators.  The  impli¬ 
cation  of  validation  is  that  the  system  will  meet  its  operational  life 
cycle  design  commitments."^  Thus  the  FDD  is  not  entirely  adequate  for 
software  maintenance,  although  it  does  contain  many  comprehensive  and 
detailed  sections. 

5 .  Program  Package  Section  1.0  Purpose,  p.  2-165. 

This  section  states:  "The  Program  Package  document  shall  con¬ 
sist  of  all  the  program  material  items  necessary  for  the  procuring  agency 
to  produce,  maintain,  and  update  the  digital  processor  progreun."  Several 
sections  of  the  PP  refer  to  card  decks  and  magnetic  tapes  as  the  media  for 
source  programs.  These  sections  should  be  augmented  to  allow  for  the 
possibility  of  other  physical  media,  such  as  disc  pack,  cassette, 
cartridge,  and  ROM.  In  addition,  the  possible  use  of  interactive  compila¬ 
tion  cind  debugging  facilities  for  software  production  and  the  storing 


Robert  0.  Lewis,  "Software  Verification  and  Validation,"  in 
John  D.  Cooper  and  Mathew  Fisher  (Editors) ,  Software  Quality  Management, 
Chapter  15 . 
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of  program  files  in  a  time-sharing  facility  should  be  recognized.  Finally, 
the  document  should  allow  for  the  possibility  of  computer  to  computer 
tr2msfer  of  program  files,  without  the  intervening  physical  media  of 
cards  and  tapes. 

6.  Operator's  Manual  Section  1.0  Purpose,  p.  4-5. 

This  section  states:  "The  Operator's  Manual  shall  be  limited  to 
instructions  for  preparing  and  maintaining  the  digital  processor  progreun 
in  the  required  state  of  capability  in  order  that  the  operational 
mission  may  be  accomplished."  As  shown  in  TABLE  2,  there  are  no  sections 
of  the  OM  that  are  considered  inadequate  for  software  maintenance. 
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TABLE  2 


SUMMARY  OP  3560.1  SECTIONS 
APPLICABLE  TO  SOFTWARE  MAINTENANCE 

INPUTS 


SEC 

SEC  # 

PAGE 

PURPOSE 

*Y/N 

REMARKS 

FOS 

3.3.1 

2-37 

CHARACTERISTICS 

N 

UNCLEAR 

PPS 

3.4.N.1 

2-95 

CHARACTERISTICS 

Y 

PDD 

3.4 

2-146 

FORMATS 

Y 

SOM 

N 

4-38 

DATA  ENTRY 

PROCESSING 

Y 

FOS 

3.3.2 

2-37 

PROGRAM  PROCESSING 

Y 

PPS 

3.4.N.2 

2-98 

FUNCTION  PROCESSING 

OUTPUTS 

Y 

FOS 

3.3.3 

2-37 

DISTRIBUTION 

Y 

PPS 

3.4.N.3 

2-99 

CHARACTERISTICS 

Y 

PDD 

3.4 

2-146 

FORMATS 

Y 

TS 

3.2.4 

3-29 

TEST  OUTPUTS 

FUNCTIONS 

V 

PPS 

3.3 

2-90 

FUNCTION  DESCRIPTION 

N 

REDUNDANT 

PPS 

3.3.5 

2-92 

FUNCTION  DESCRIPTION 

N 

REDUNDANT 

PPS 

3.4 

2-95 

FUNCTION  REQUIREMENT 

N 

REDUNDANT 

PPS 

TABLE  3-11 

2-97 

FUNCTION  VS.  STATE 

Y 

GOOD  TABLE 

PDS 

3.1 

2-123 

FUNCTION  REQUIREMENT 

Y 

PDS 

3.2 

2-124 

FUNCTION  DESCRIPTION 

Y 

PDS 

3.4 

2-127 

FUNCTION  FLOW 

Y 

*Y/N  COLUMN;  ”Y”  MEANS  SECTION  IS  ADEQUATE  FOR  SOFTWARE,  MAINTENANCE. 

"N"  MEANS  SECTION  IS  INADEQUATE  FOR  SOFTWARE  MAINTENANCE. 
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DATABASE 


SEC 

SEC  # 

PAGE 

PURPOSE 

Y/N 

REMARKS 

PPS 

3.5 

2-99 

REQUIREMENTS 

N 

UNCLEAR 

PDD 

3.3.2 

2-145 

SUBPROGRAM  DATA  DESIGN 

N 

TERMINOLOGY 

DBO 

ALL 

2-153 

COMMON  DATA  DESCRIPTION 

INTERFACES 

Y 

SOS 

5.2 

2-14 

PERIPHERAL  SYSTEM 

N 

VAGUE 

SOS 

5.3 

2-15 

OPERATOR 

N 

VAGUE 

SOS 

5.4 

2-15 

INTERSYSTEM 

Y 

SOD 

5.2 

2-28 

PERIPHERAL  SYSTEM 

Y 

SOD 

5.3 

2-28 

OPERATOR 

N 

INCOMPLETE 

SOD 

5.4 

2-28 

INTERSYSTEM  ON-LINE 

Y 

IDS 

1.0 

2-41 

INTERFACE  REQUIREMENTS 

N 

CONFUSING 

IDS 

2. 0-8. 3 

2-42 

INTERFACE  REQUIREMENTS 

Y 

GOOD  SECTIONS 

PPS 

3.2.3 

2-89 

INTERFACE  I.D. 

Y 

PPS 

3.3.3 

2-92 

DIG.  PROC.  BLOCK  DIA. 

Y 

PPS 

3.3.4 

2-92 

PROGRAM 

Y 

PDD 

3.8 

2-149 

SUBPROGRAMS 

Y 

PDD 

PIG.  3-2 

2-150 

SUBPROGRAMS 

TESTING 

Y 

PPS 

3.4.N.4 

2-99 

SPECIAL  REQUIREMENTS 

Y 

PPS 

4.2 

2-101 

TEST  REQUIREMENTS 

Y 

FOD 

5 

2-115 

TEST  &  SIM.  SCENARIOS 

Y 

TPL 

1.0 

3-5 

SCOPE  S  LEVELS 

N 

TERMINOLOGY 
VALIDATION  NEEDED 

TPL 

1.1.2 

3-7 

EQUIPMENT 

Y 

TPL 

1.1.3 

3-8 

SUPPORT  SOFTWARE 

Y 

TS 

3.2 

3-23 

SYS. /PROG.  DEFINITION 

Y 

TPR 

1.0 

3-35 

INSTRUCTIONS  S  EVAL. 

N 

UNCLEAR 

TPR 

3.2.1 

3-40 

EQUIPMENT  PREPARATION 

Y 

TPR 

3.2.3 

3-41 

TEST  PROCEDURE 

N 

UNCLEAR 

TPR 

7 

3-43 

SUPPORT  SOFT.  REQUIRE. 

Y 

TR 

ALL 

3-45 

TEST  REPORTS 

Y 

EXCEPT 

PATCH  REFERENCE 
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QUALITY  ASSURANCE 


SEC 

SEC  # 

PAPE 

PURPOSE 

Y/N 

REMARKS 

PPS 

4. 

2-100 

QA  PROVISIONS 

N 

VALIDATION  NEEDED 

PDD 

4. 

2-149 

QA  PROVISIONS 

N 

VALIDATION  NEEDED 

TPL 

9. 

3-14 

EVALUATE  TEST  RESULTS 

N 

LOOSE 

TS 

9. 

3-26 

SPECIFY  TEST  REPORTS 

N 

VALIDATION  NEEDED 

SUBPROGRAMS  &  SUBROUTINES 

PDS 

3.4.3 

2-131 

REFERENCE  CONTROL 

Y 

PDS 

6. 

2-133 

COMMON  SUBROUTINES 

Y 

PDD 

1.0 

2-137 

SUBPROGRAM  STRUCTURE 

Y 

PDD 

3.5 

2-148 

LIBRARY  SUBROUTINES 

Y 

PDD 

3.-6 

2-148 

INITIATION 

Y 

PROGRAMMING 

SOD 

6.1 

2-29 

STANDARDS 

y 

PDS 

3.3 

2-127/8 

MEM.  S  PROC.  ALLOC. 

Y 

PDS 

3.5 

2-131 

LANGUAGE 

Y 

CAPACITY  REQUIREMENTS 

FOS 

5.2.2 

2-39 

CONSOLE  CAPACITY 

Y 

PPS 

3.5 

2-100 

PROGRAM  CAPACITY 

Y 

CONFIGURATION  MANAGEMENT 

SOS 

4.2 

2-14 

PHYSICAL  CONFIGURATION 

Y 

PDS 

3.5 

2-132 

PROGRAM  CONFIGURATION 

y 

OM 

3.1 

4-8 

MINIMUM  CONFIGURATION 

Y 

PDM 

3 

4-17 

SUBPROGRAM  CONFIGURATION 

Y 

OPERATION 

OM 

4.3 

4-9 

PARAMETER  VALUES 

Y 

OM 

6 

4-9 

TROUBLE  REPORTS 

Y 

OM 

7 

4-10 

RECOVERY 

Y 
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INPUTS 


Section  3.3.1  (p.  2-37)  of  FOS  reads:  "Data  type  by  source,  its  period¬ 
icity  of  update  rate,  cind  expected  and/or  reliability  will  be  provided 
in  this  paragraph."  The  meaning  of  the  underlined  words  is  unclear. 


FUNCTIONS 

Sections  3.3  (p.  2-90),  3.3.5  (p.  2-92)  and  3.4  (p.  2-95)  of  PPS  overlap 
to  some  extent. 


DATABASE 

Section  3.5  (p.  2-99)  of  PPS  reads:  "Adaptation  data  is  that  data  that 
can  be  centrally  modified  as  needed  to  define  the  scope  of  operational 
functions  within  prescribed  limits."  The  meaning  of  this  statement  is 
unclear . 

Section  3.3.2  (p.  2-145),  Variables  of  PDD  refers  to  "program"  in  line  2 
and  "constauit"  under  "a.  constant  name.”  The  word  "variable"  was 
probably  intended  in  both  cases. 


INTERFACES 

Sections  5.2  (p.  2-14)  and  5.3  (p.  2-15)  of  SOS  provide  insufficient 
information  concerning  what  comprises  a  peripheral  systems  interface 
and  operator  interface,  respectively. 

Section  5.3  (p.  2-28)  of  SOD  references  a  Peripheral  System  Interface 
section  5.2  rather  than  spelling  out  what  is  requred  for  the  Operator 
Interface . 
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IHTERFACES  (CONTINUED) 


Section  1.0  (p.2-41)  of  IDS  reads:  "This  specification  establishes  a  set 
of  requirements  for  the  preparation  of  a  document  for  defining  the  design 
interdiqital  processor  digital  interfaces."  The  underlined  portion  is 
confusing. 


TESTING 

Section  1.0  (p.  3-5)  of  TPL  reads:  "The  test  plan  shall  define  the  scope 
of  tests  required  to  ensure  that  the  system,  function,  and/or  progreim 
meets  all  applicable  technical ,  operational  and  performance  specification. 
Since  the  only  categories  of  speciation  in  3560.1  are  "operational," 
"performance,"  and  "design,"  the  meaning  of  "technical"  is  not  cleeur. 

Section  1.0c  (p.  3-6)  of  TPL  (Function  Test)  and  Section  l.Od  (p.  3-6) 
should  stipulate  validation  against  the  TOR  in  addition  to  verification 
against  performance  and  design  specifications  and  program  description 
document,  respectively. 

Section  1.0  (p.  3-35)  of  TPR  reads:  "Test  procedures  provide  for  the 
quantitative  results  of  tests,  which  are  later  extracted  for  the  tests 
themselves."  The  meaning  of  this  statement  is  not  clear. 

Sections  3.2. 3 j  (p.  3-42)  of  TPR  reads:  "The  procedure  must  agree 
exactly  with  the  program  behavior."  The  meaming  of  this  statement  is 
not  clear. 
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TESTING  (CONTINUED) 


Sections  1.1.1  (p.  3-45)  and  1.  (p.  3-49)  of  TR  imply  the  allowance  of 
patches  in  programs  during  testing.  This  is  believed  to  be  a  poor 
practice . 


QUALITY  ASSURANCE 

Section  4.  (p.  2-100)  of  PPS,  Section  4.  (p.  2-149)  of  PDD,  and  Section 
9.  (p.  3-26)  of  TS  should  stipulate  validation  against  the  TOR  in  addition 
to  program  verification. 

Section  9.  (p.  3-14)  of  TPL  should  contain  a  statement  that  government 
personnel  must  witnes.s  the  tests.  Although  this  section,  as  written,  is 
concerned  with  procurement  rather  than  maintenance,  the  argument  for 
using  government  witnesses  is  valid  for  maintenance. 
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I 

I 

V.  CONCLUSIONS  AND  ^COH’JSNDATIONS 

Based  on  the  foregoing  analysis,  the  following  conclusions  and 
recommendations  are  presented  relative  to  the  usability  of  3560.1  for 
software  maintenance . 

1.  The  standard  is  comprehensive  and  detailed.  Considering  the  fact 
that  it  was  issued  in  1974,  it  was  unable  to  benefit  from  the  hindsight 
that  is  expressed  in  this  report,  which  is  based  mainly  on  advances  that 
have  been  made  in  programming  .methodology  since  1974,  If  3560.1  were 
updated  to  reflect,  new  programming  technology,  it  could  be  a  more  com¬ 
prehensive  standeird  than  MIL-STD  1679.  Notaible  aspects  of  the  standcird 
are  the  following: 

-  Applicable  Documents  statements. 

-  Resource  budgets  (time,  space) . 

-  Numerous  exaunples. 

-  Content  Check  Lists. 

Interfaces  descriptions. 

-  Test  coverage . 

-  Detailed  Table  of  Contents  for  each  specification. 

2.  As  pointed  out  in  Section  III,  there  au:e  some  deficiences  in  the 
vital  area  of  traceability. 

3.  As  demonstrated  in  Section  IV,  there  should  be  more  emphasis  on 
validation.  To  quote  from  page  14  of  3560.1:  "The  Tactical  Operational 
Requirement  shall  serve  as  a  life  cycle  configuration  management  device 
for  specifying  the  overall  tactical  operational  software  capability 
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requirements  for  the  combat  system."  That  being  the  case,  there  should 
be  more  emphasis  on  validating  against  the  TOR,  with  respect  to  QA 
procedures,  during  both  development  and  maintenance. 

4.  There  are  many  examples  of  redundancies  and  use  of  inconsistent 
terminology  in  the  standard.  Examples  of  the  former  are: 

-  Section  7.  Data  Unit  Descriptions,  p.  2-57  and  Section  8. 
Message  Descriptions,  p.  2-69  of  Interface  Design  Specification 
contain  similar  material. 

Test  specification  System,  pp.  3-20  to  3-26  and  Test  Specifica¬ 
tion  Fvmction,  pp.  3-27  to  3-31  contain  similar  material. 

Much  of  the  material  in  the  Test  Plans  and  Test  Specifications 
is  similar.  These  could  be  combined  into  a  single  document, 
with  resultant  reduction  in  verbage  and  increase  in  clarity. 

-  Each  of  the  documents  is  described  in  the  format  of  purpose , 
scope,  t^'pical  content,  etc.  followed  by  the  actual  format  of 
document  content.  Much  of  the  material  is  duplicated  between 
the  two  sections  (e.g..  Test  Procedures  pp.  3-35  to  3-37  and 
pp.  3-39  to  3-42).  This  material  could  be  combined  in  many 
of  the  documents . 

Examples  of  inconsistent  terminology  are; 

-  Section  1.0  Purpose,  p.  2-153  and  Section  9.  Notes,  p.  2-162 

of  Data  Base  Design  refer  to  a  Subprogram  Description  Document. 
This  document  is  not  defined  or  described  in  3560.1  by  that 
name . 

-  Section  1.1.3  Analysis,  p.  3-46  of  Test  Reports  refers  to 
Operational/Functional  Specification.  This  specification  is 
not  defined  or  described  in  3560.1  by  that  name. 


5.  The  standard  is  much  more  oriented  to  software  development  them 
to  software  mainteneince .  It  also  seems  to  have  a  strong  orientation  to 
the  Navy  Tactical  Data  System.  A  more  general  orientation  might  be 
preferable  in  order  to  achieve  wider  applicability  to  a  variety  of 
software  systems . 

6.  It  would  be  useful  for  both  softw2ure  development  and  maintenance 
activities  to  provide  a  section  in  the  standard  that  describes  the 
material  in  subject  matter  instead  of  document  format,  i.e.,  a  descrip¬ 
tion  of  all  document  sections  that  refer  to  inputs ,  all  that  refer  to 
outputs,  etc.  Such  a  breakdown  was  used  in  TABLE  2  of  this  report. 

7 .  In  summary ,  an  inexperienced  programmer  would  probably  have 
difficulty  applying  3560.1  to  maintenance  because  of  the  problems 
mentioned  in  this  report.  Because  of  the  great  shortage  of  skilled 
software  personnel,  the  criterion  of  usability  of  the  standard  by  an 
inexperienced  programmer  is  appropriate. 
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